One Size Doesn’t Fit All 

Customizing Educational Technology Professional Development 


By Judi Harris 

Part Four: 

Evaluating ETPD Designs 

Previous articles reviewed the 
range of educational technol- 
ogy professional development 
program goals, explained 
various ETPD models, and 
addressed how to combine 
goals and models to fit partic- 
ular teachers’ characteristics. 
Now we’ll assess the efficacy 
of those designs. 


E ducational technology-related 
professional development can 
be designed in many differ- 
ent ways. ETPD varies by general 
purposes and goals, specific learn- 
ing objectives, curriculum content, 
student grade levels for which the 
strategies and tools presented are ap- 
propriate, professional development 
models used, how it is matched to 
participating teachers’ characteristics, 
and the ways in which it is evaluated. 
Providers can ensure the effectiveness 
of technology-related professional 
development by designing ETPD ses- 
sions and programs with these seven 
aspects in mind. 

Last month’s article in this series 
discussed how to combine ETPD 
goals and models, matching these 
combinations to fit particular teachers’ 
characteristics and needs. This month, 
we will address how to determine the 
efficacy of ETPD designs for the edu- 
cators for whom they were created. 

ETPD Goals, Models, and Matching 

There are six general goals that ETPD 
sessions or programs can address, 
either singly or in combination: 

• Awareness and/or trial of specific 
tools or resources 

• Curriculum integration in specific 
content areas 

• Change in instructional practice, 
focusing on specific instructional 
techniques 

• Curriculum and/or instructional 
reform 

• School organizational or cultural 
change 

• Social change beyond the school 


There are 20 different ETPD in- 
structional models, classified into five 
general types according to the kinds of 
professional learning that character- 
ize each. ( Editor’s note: For a detailed 
description of the various models, see 
Part Two of this series in the March/ 
April issue of L&L.) Examples of spe- 
cific ETPD programs that illustrate 
the goals and models are linked on the 
ETPD Web site. See the table below. 

One of the keys to effective ETPD 
design is to match models to goals and 
both to participating teachers’ needs, 
preferences, and characteristics. The 
following combinations, though not 
an exhaustive list, illustrate effective 
matches. 


Example Combinations of ETPD Goals, Models, 

Session or Program Goals 

Awareness/trial of specific tools or resources 


Curriculum integration in specific content areas 


Change in instructional practice (specific techniques) 


Curriculum and/or instructional reform 

School organizational or cultural change 
Social change beyond the school 
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Aligning ETPD Evaluation 
The success of a particular ETPD 
design is determined by how well its 
content and structure are aligned with 
participating teachers’ professional 
learning needs, interests, preferences, 
and contextual realities. Evaluating 
ETPD designs must be done accord- 
ing to the goals that were selected, 
and within the professional contexts 
for which they were created. 

If a goal for an ETPD program is 
use of a range of digital technologies 
in teachers’ classrooms in a particular 
content area — physical science, for 
example — it would be inappropriate 
to evaluate the success of the program 
based only on teachers’ stated knowl- 
edge of technology integration options 
in physical science. If a goal of the pro- 
gram is curriculum integration, then 
evaluators should seek evidence of that 
integration in participants’ classrooms. 


and Teacher Characteristics 



Professional Development Models 

Teacher Characteristics (Adopter Categories) 

Demonstration or awareness sessions 

Early majority, late majority 

Hands-on workshops 

Early majority, late majority 

Unassisted independent exploration 

Innovators 

Assisted individualized exploration 

Latest adopters 

Individualized, learner-created learning plan 

Innovators 

Prescribed and managed individualized instruction 

Latest adopters 

Peer-to-peer classroom visits 

Early adopters, early majority, late majority 

Sharing best practices 

Early adopters, early majority 

Peer coaching 

Early adopters, early majority, late majority 

Large- and small-group problem solving 

Early adopters, early majority 

Collaborative learning (five models) 

Lesson study 

Early adopters, early majority, late majority 

Mentoring 

Early adopters, early majority 

Action research done with other teachers 

Early adopters 

Collaborative creation of materials & approaches 

Early adopters, early majority 

Individual creation of materials & approaches 

Innovators, early adopters 

Individual, collaborative, and/or externally assisted action research 

Innovators, early adopters 

All models, emphasizing all forms of cooperation & collaboration 

All adopter categories, involved in different activities at different times 

All models 

All adopter categories, involved in different activities at different times 


Copyright © 2008, ISTE (International Society for Technology in Education), 1 .800.336.5 1 9 1 (U.S. & Canada) or 1 .541 .302.3777 (Inti), iste@iste.org, www.iste.org.AII rights reserved. 

June/July 2008 I Learning & Leading with Technology 25 







































Evaluating ETPD designs is primarily a practical analysis of how well they fit the learning needs 
and preferences of participating teachers. This is different from assessing those teachers’ developing 
educational technology knowledge. 


Unfortunately, the majority of K-12 
ETPD evaluation data generated and 
analyzed to date do not correlate di- 
rectly with the stated goals of ETPD 
sessions and programs. Most of what 
is requested is teachers’ attitudes 
about and opinions of ETPD experi- 
ences. Although this information is 
helpful in evaluating ETPD designs’ 
appeal, by itself it does not provide 
us with enough of the right kinds of 
information to help us to evaluate the 
efficacy of the design. In evaluating it, 
we seek to understand how to design 
the next session or program for the 
same group of teachers, or how to re- 
design the current one for use with a 
different group. 

Evaluating ETPD designs, then, is 
primarily a practical analysis of how 
well they fit the learning needs and pref- 
erences of participating teachers. This 
is different from assessing those teach- 
ers’ developing educational technology 
knowledge. Assessment is an ongoing 
process in which participants’ learning 
and progress are determined forma- 
tively, as ETPD continues. It is similar 
in process and aims to the assessment 
of students’ curriculum-based learning. 
Evaluating ETPD designs is done only 
partially based on assessment. 

How can we evaluate ETPD de- 
signs? Remembering the importance 
of matching what we evaluate to the 
stated goals of the ETPD session or 
program being evaluated, there are 
multiple methods from which to 
choose. 

ETPD Evaluation Methods 
Evaluating ETPD seeks answers to 
the question, “How well did it work?” 
Though that question seems simple, 
on closer inspection it is fairly com- 
plex. Valid answers can be obtained by 
seeking out participating teachers’: 


• Perceptions of the ETPD session or 
program 

• Attitudes toward the ETPD’s focus 

• Knowledge of the ETPD’s focus 

• Intentions/ decisions to use what they 
learned in the ETPD 

• Use of what they learned in their 
professional practice 

• Students’ learning when using what 
participating teachers learned in the 
ETPD 

There are several ways to find an- 
swers that investigate each type of 
evaluation information. 

Perceptions of the ETPD session or 
program — how well it was organized, 
taught, and supported — are most often 
gathered with a brief survey of partici- 
pants, done either on paper or online 
after the session or program is com- 
plete. Group discussions with partici- 
pants (sometimes called focus group 
interviews) are also useful for gather- 
ing perceptions. So that participants 
feel free to express all aspects of their 
perceptions, it’s best if surveys are com- 
pleted anonymously and if focus group 
interviews are facilitated by someone 
other than the ETPD instructors. 

Attitudes toward the ETPD’s focus- 
perceptions of the utility, complexity, 
and educational appropriateness of 
the tools, resources, or instructional 
strategies that the ETPD focused on — 
are also most often gathered using 
a survey. Individual interviews with 
participants help generate more in- 
depth information on the reasons for 
these attitudes. Again, it’s best to gen- 
erate survey data anonymously and 
for someone other than the ETPD’s 
instructors to talk with participants 
individually. A powerful way to evalu- 
ate the success of the ETPD is to ask 
participants to complete the survey or 
participate in an interview both before 
and after being involved in the ETPD 
session or program. 


Using before-and-after surveys (e.g., 
quizzes) or interviews also helps as- 
certain participants’ knowledge of the 
ETPD’s focus. A more unusual — but 
very effective — way to understand 
participants’ knowledge is to observe 
them sharing what they have learned 
with colleagues who were not involved 
directly in the ETPD session or pro- 
gram, but who want to learn about the 
ETPD’s focus. 

Brief, anonymous surveys completed 
after participation helps with discover- 
ing participants’ intentions/ decisions 
to use what they learned in the ETPD. 
The items simply ask how likely it is 
that each tool, resource, or technique 
shared in the ETPD will be used in the 
teacher’s practice, when that is likely to 
occur, and why. A pair of items like the 
examples below should be included 
that addresses each of the tools, re- 
sources and/or techniques that were 
shared in the ETPD. 

• How likely is it that you will use 
in your teaching? 

° Almost certainly 
° Very likely 
° Likely 

° Somewhat likely 
° Not very likely 

° Please tell us why you responded 
in this way: 

• When do you think that you will 

use in your teaching? 

° As soon as possible 

° Soon 

° After some time has passed 
° Eventually 
° Probably never 

° Please tell us why you responded 
in this way: 

Survey or interview self-reports 
similar to those described above help 
ascertain participating teachers’ use of 
what they learned, but most research- 
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Evaluation Information Matched to ETPD Goals 


Session or Program Goals 

Evaluation Information Sought 

Awareness and/or trial 

Perceptions 

of specific tools or resources 

Attitudes 

Knowledge 

Curriculum integration 

Perceptions 

in specific content areas 

Knowledge 

Intentions/decisions 

Use 

Students’ learning 

Change in instructional practice, focusing on 

Perceptions 

specific instructional techniques 

Attitudes 

Knowledge 

Intentions/decisions 

Use 

Students’ learning 

Curriculum and/or instructional reform 

Perceptions 

Attitudes 

Knowledge 

Intentions/decisions 

Use 

Students’ learning 

School organizational or cultural change 

Perceptions 

Intentions/decisions 

Use 

Students’ learning (only if it relates directly to 
school organizational or cultural change) 

Social change beyond the school 

Perceptions 

Intentions/decisions 

Use 


ers suggest that we should not rely on 
self-reports alone for evaluating use of 
ETPD content. Instead, evaluaters can 
analyze teachers’ lesson plans, records 
of team instructional planning ses- 
sions, and observations of teaching. 
ETPD providers should be careful, 
though, not to abandon their collegial 
roles when doing these kinds of evalu- 
ations. Observations and analyses of 
planning documents should be done 
as much as possible in a peer-to-peer, 
constructively critical, and collaborative 
way, so that participating teachers are 
not seeing preparing observed lessons 
or documents as performances that are 
different from their everyday practice. 

Though it is ultimately the end goal 
for all ETPD, examining evidence 
of students’ learning that is done us- 
ing tools, resources, and techniques 
shared in the ETPD session or pro- 
gram is both complex and challeng- 
ing. Because the classroom is a com- 
plex context, with many factors and 
conditions affecting students’ learning 
(and teachers’ teaching), it is probably 
impossible to determine clear cause 
and effect between something learned 
in ETPD, then applied in a classroom. 
And yet, analyzing students’ work 
done before and after new practices 
related to an ETPD session or pro- 
gram have been implemented can help 
us to gauge impact, as can interviews 
with older students — usually done in 
groups — that inquire about their per- 
ceptions of the quality and efficacy of 
their learning using the new practices. 

Matching Goals to Information Sought 
As stated earlier, evaluation of ETPD 
must be keyed directly to its goals. 
Which evaluation information helps 
us to evaluate which ETPD goals? 

The Table above displays probable 
matches, with “best match” evaluation 
information types underlined. 


Note that participants’ perceptions 
of how well the ETPD session or 
program was organized, taught, and 
supported, and their intentions or de- 
cisions to use what they learned in the 
ETPD are the two types of evaluation 
data that are appropriate to seek for 
all six possible ETPD goals. Though 
it could be argued that all types of 
information are potentially helpful in 
evaluating ETPD designed to address 
all goals, ETPD that seeks to assist 
with change in instructional practice 
and curriculum or instructional reform 
are the types that are best evaluated 
using the full range of ETPD evalua- 
tion information. 

Considering all of what we know 
about professional development, and 
the arguably greater amount of what 


we still have to learn about it, evalua- 
tion is by far our largest professional 
and developmental learning oppor- 
tunity within ETPD. Examples and 
descriptions of ETPD evaluations are 
available online for your perusal at the 
ETPD Web site. 

Resources 

Designing Educational Technology Professional 
Development Web site: http://etpd.wm.edu 
Lawless, K. A., & Pellegrino, J. W. (2007). 
Professional development in integrating 
technology into teaching and learning: 
Knowns, unknowns, and ways to pursue 
better questions and answers. Review of 
Educational Research, 77, 575-614. 

Judi Harris is a professor and the 
Pavey Family Chair in Educa- 
tional Technology in the School 
of Education at the College of 
William & Mary in Williams- 
burg, Virginia, USA. Her work 
focuses on professional develop- 
ment for teachers in curriculum-based technology 
integration and via online mentoring. 


It’s best to generate survey data anonymously and for someone other 
than the ETPD’s instructors to talk with participants individually. 
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